Skip to content

Deallocate the GPU pool's idle node so later runs find the PyTorch image - #54

Merged
paolosalvatori merged 2 commits into
mainfrom
gpu-node-pools-image-download
Nov 9, 2026
Merged

Deallocate the GPU pool's idle node so later runs find the PyTorch image#54
paolosalvatori merged 2 commits into
mainfrom
gpu-node-pools-image-download

Conversation

@paolosalvatori

@paolosalvatori paolosalvatori commented Nov 9, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Motivation

Every run of the training job in the GPU node pools tutorial waited 5 to 6 minutes before its pod started, though the note under How to run it said that happened only the first time.
The GPU node downloaded pytorch/pytorch:2.5.1-cuda12.4-cudnn9-runtime (about 3 GB) again on every run, because the cluster autoscaler deleted the idle node, and the image with it, once the previous job was done.
06-simulate-eviction.sh waited twice. For a demo, that is too long.

AKS keeps a node's images through a scale-down when the pool's Scale-down Mode is Deallocate.
The cluster autoscaler then deallocates the idle node instead of deleting it, and the node's OS disk keeps the images its pods pulled.
The next time a pod needs the pool, the autoscaler starts the node again, before it adds any new node.
Spot pools have supported it since it became generally available, and it requires a managed OS disk.

Changes

  • scripts/00-variables.sh: adds GPU_NODE_SCALE_DOWN_MODE="Deallocate" and GPU_NODE_OS_DISK_TYPE="Managed", with a comment on what the mode does and why an eviction still deletes the node.
  • scripts/01-add-gpu-node-pool.sh:
    • Adds the pool with --scale-down-mode and --node-osdisk-type.
    • An existing pool's OS disk type, which can't change, joins the VM size and priority it must match.
    • An existing pool gets the scale-down mode with az aks nodepool update --scale-down-mode.
    • The printed pool shows osDiskType and scaleDownMode.
  • scripts/05-run-training-job.sh and scripts/06-simulate-eviction.sh: their comments and messages say that:
    • a run may start the node an earlier run left deallocated;
    • an eviction deletes its node, because the pool's eviction policy (Delete) applies to it rather than its scale-down mode, so the node that replaces it pulls the image again.
  • README.md:
    • The note now says that the first run in a cluster waits for the download and the next runs don't, except on the node that replaces the evicted one.
    • A new A node that keeps its images paragraph explains the mode, its cost (the OS disk, and a deallocated Spot node still counts against the Spot vCPU quota) and why the OS disk is managed.
    • The scripts table, the eviction paragraph and the resources follow.

The first commit corrected the note for the emulator as it is today; the second rewrites it for the new behaviour.

Tests

  • Tested on LocalStack for Azure run from the source of localstack/localstack-pro pull request 10317, on a new cluster created by scripts/01-user-assigned-managed-identity.sh:
    • Setup: 01 to 04 passed. The pool reports osDiskType: Managed, scaleDownMode: Deallocate, scaleSetPriority: Spot and scaleSetEvictionPolicy: Delete.
    • First 05-run-training-job.sh: 423 s, of which 5m20.526s were the new node's image pull. 35 s after the job ended, the autoscaler deallocated the node: its container exited, the Node went NotReady with the Cloud Controller Manager's shutdown taint, the VMSS instance reported PowerState/deallocated, and the pool reported count: 0.
    • 06-simulate-eviction.sh: passed in 657 s. Its pod started on the restarted node 31 s after its creation, with Container image "pytorch/pytorch:2.5.1-cuda12.4-cudnn9-runtime" already present on machine. The eviction deleted that node, and the replacement pod's new node pulled the image in 5m53.577s.
    • Second 05-run-training-job.sh: 154 s, on the restarted node, with the image already present.
  • On localstack/localstack-azure:latest (build 2026.11.0.dev770), before this change, a later run's pod spent 5m12.656s pulling the image. Until pull request 10317 ships, the emulator accepts --scale-down-mode Deallocate but deletes the idle node as before.
  • bash -n passes on the changed scripts. pre-commit run: gitleaks passed, zizmor skipped. The note renders as a Note alert through GitHub's markdown API.
  • Not run on Azure: adding a Spot Standard_NV18ads_A10_v5 pool with --scale-down-mode Deallocate --node-osdisk-type Managed. Microsoft Learn documents both flags, and Scale-down Mode's support for Spot pools.

Related

  • localstack/localstack-pro pull request 10317: the emulator's implementation of Scale-down Mode Deallocate. This pull request should merge once that one ships in localstack/localstack-azure:latest.

🤖 Generated with Claude Code

https://claude.ai/code/session_01FvNuJKW5UZQ1Rce8qnG2GR

@paolosalvatori @claude
Say that every run of the GPU training job waits for its PyTorch image
e471c5f 
The note under "How to run it" said that the job's pod waits for its image the first time you run the tutorial, but it waits on every run.
Each node the cluster autoscaler adds to the GPU pool is a new k3d container with an image store of its own, and the emulator's autoscaler removes a node, and its images with it, once the node has been idle for 30 seconds, so no run finds the image already downloaded.
On a later run on LocalStack for Azure, the kubelet again logged "Successfully pulled image ... in 5m12.656s".
The note now says that every run waits and why, and that 06-simulate-eviction.sh waits twice.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FvNuJKW5UZQ1Rce8qnG2GR
Copilot AI balanced review requested due to automatic review settings November 9, 2026 09:02
Copilot AI reviewed Nov 9, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The tutorial overstates image-download requirements for reruns and should qualify those claims to account for cached-image reuse when no fresh GPU node is needed.

1 open issue
What changed in this PR

This documentation-only PR updates the GPU node pools tutorial to explain recurring PyTorch image download delays as GPU nodes are removed and recreated.

Changes:

  • Replaces the first-run warning with recurring download expectations for training and eviction scripts.
  • Explains node-local image caching and the different idle scale-down delays in LocalStack and AKS.
File Description
tutorials/​gpu-node-pools/​README.md Expands the startup-delay note with image lifecycle and script-specific download expectations.

Review effort: Balanced

Comment thread tutorials/gpu-node-pools/README.md Resolved
@paolosalvatori @claude
Deallocate the GPU pool's idle node so later runs find the PyTorch image
5e4c9d6 
Each run of a training job waited 5 to 6 minutes for its GPU node to pull pytorch/pytorch:2.5.1-cuda12.4-cudnn9-runtime, because the cluster autoscaler deleted the idle node, and the image with it, once the job was done.
01-add-gpu-node-pool.sh now adds the GPU pool with --scale-down-mode Deallocate and --node-osdisk-type Managed: the autoscaler deallocates the idle node instead, its OS disk keeps the image, and it starts that node again, before it adds any new node, for the next job (Microsoft Learn, aks/scale-down-mode, which supports Spot pools since its general availability and requires a managed OS disk).
A pool that already exists gets the scale-down mode with az aks nodepool update; its OS disk type can't change, so a pool with another one stops the script, as one with another VM size or priority already did.
The eviction in 06-simulate-eviction.sh still deletes its node, under the pool's Delete eviction policy, so the node that replaces it pulls the image again.
The README's note, the scripts table and "How it works" say so, and 05 and 06 describe the node a run starts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FvNuJKW5UZQ1Rce8qnG2GR
@paolosalvatori paolosalvatori changed the title Say that every run of the GPU training job waits for its PyTorch image Deallocate the GPU pool's idle node so later runs find the PyTorch image Nov 9, 2026
@paolosalvatori
paolosalvatori merged commit 5e8bcba into main Nov 9, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@paolosalvatori